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Abstract 


This  project  brought  together  elements  of  virtual  reality  technology  (immersive 
technology)  and  elements  of  scientific  visualization  to  address  issues  of  high  dimensional 
spatial  and  volumetric  visualization  problems.  Applications  of  such  methods  were  to  the 
visualization  of  any  spatially  or  volumetrically  distributed  data  with  multivariate  data 
characterization  at  each  pixel  or  voxel.  A  military  application  was,  for  example, 
characterization  of  mine  fields.  This  research  accomplished  under  this  funding  has  a  very 
strong  dual  usage  character. 

We  developed  several  visualization  methodologies  based  on  the  concept  of  a 
mathematical  grand  tour  and  based  on  recursive,  data-adaptive  pattern  recognition.  These 
visualization  methods  are  embedded  in  immersive  display  systems  which  include  a  head- 
mounted  display  system  and  a  stereographic  projection  system.  Research  included  tasks 
oriented  toward  developing  approximation  models  for  general  two-  and  three- 
dimensional  geometric  structures,  visualizing  spatial  and  multivariate  multivariate  data, 
integrating  high  performance  parallel  and  graphics  computers,  and  developing 
psychometric  comparisons  for  stereo  visualization. 

Computer  and  virtual  reality  resources  were  largely  in  place  so  that  there  was 
little  or  no  latency  in  the  development  of  this  project.  An  added  bonus  is  the  leverage 
obtained  with  the  impending  installation  of  an  Intel  Paragon  supercomputer  at  George 
Mason  University.  This  project  was  very  effective  producing  26  technical  reports 
including  two  Ph.D.  dissertations  and  30  published  papers. 


Introduction 

This  proposal  focused  on  the  alliance  of  elements  of  virtual  reality  technology  and 
elements  of  scientific  visualization  to  address  issues  of  mine  detection  and  related  spatial 
and  volumetric  visualization  problems.  By  virtual  reality  or  virtual  environments,  we 
mean  an  immersive  visual  and  audio  technology  such  that  experimenter  has  little  or  no 
awareness  of  the  real  environment.  For  our  purposes  of  data  visualization,  this  is  intended 
as  a  focusing  device  so  that  the  experimenter  has  a  heightened  sense  of  awareness  of  the 
problem  at  hand  and,  thus,  can  concentrate  in  a  natural  way  his  or  her  full  mental 
resources.  We  believe  that  this  singular  focus  is  particularly  valuable  in  adversarial 
settings  in  which  human  lives  are  at  stake.  Moreover,  we  believe  that  the  immersive 
technology,  because  of  its  three  dimensional  aspects,  will  be  less  fatiguing  than  other 
types  of  human-computer  interfaces,  current  technology  limitations  notwithstanding. 

Much  of  currently  fashionable  work  on  scientific  visualization  has  been  focused  on 
rendering  of  flow  fields  arising  from  combustion  or  meteorological  applications, 
molecular,  atomic  or  subatomic  particle  dynamics,  and  other  settings  modeled  with 
partial  differential  equation  models.  Our  focus  had  been,  in  contrast,  on  data 
representation,  exploratory  data  analysis  and  model  building  using  high  performance 
computer  graphics.  It  was  our  intention  to  continue  the  development  of  our  graphics  and 
visualization  tools  in  the  context  of  immersive  technology. 


Virtual  Reality,  Immersive  Technology  and  ScientiHc  Visualization 

The  development  of  virtual  reality  as  a  graphics  construct  began  in  the  mid¬ 
seventies  with  attempts  to  develop  more  realistic  flight  simulation.  The  standard 
paradigm  was  to  have  training  pilots  look  at  a  visual  display  screen.  The  effect,  as  might 
be  expected,  was  that  these  pilots  were  aware  that  they  were  looking  at  a  visual  display 
screen  and,  consequently,  had  a  comparatively  strong  sense  of  unreality  about  the  whole 
exercise.  An  approach  to  overcoming  this  problem  has  become  known  as  virtual  reality. 
The  intent  of  the  virtual  reality  construct  is  to  create  a  more  intimate  sense  of 
involvement  (originally  in  the  flight  simulation),  i.e.  a  sense  of  immersion.  Virtual  reality 
constructs  can  be  implemented  in  several  ways.  The  goal  is  to  recreate  via  computer  the 
items  that  the  subject's  senses  might  sense  in  reality.  Visually,  this  would  mean  replacing 
a  flat  display  screen  with  binocular  stereo  views  which  would  change  along  with  the 
motion  of  the  subject's  head.  That  is,  if  the  subject  looked  down,  he  or  she  would  see  the 
view  which  is  below.  If  the  subject  looked  backwards,  he  or  she  would  see  the  view  in 
back.  This  is  accomplished  with  a  headgear  containing  miniature  binocular  video 
displays.  There  is  an  audio  component  which  can  be  achieved  in  the  same  headgear  by 
incorporating  binaural  headphones.  Finally,  one  would  want  to  create  a  tactile  component 
which  can  be  achieved  by  incorporating  instrumented  gloves.  The  technology  is  now 
becoming  conamercially  available,  but  has  been  demonstrated  for  several  applications 
including  the  flight  simulation  application  as  well  as  for  an  interesting  application  giving 
a  sense  of  liberation  to  handicapped  individuals. 

The  major  new  thrust  in  statistical  data  analysis  and  visualization  is  to  combine 
scientific  visualization,  exploratory  data  analysis  and  virtual  reality  into  a  new  technology 
for  exploring  data.  There  is  a  history  of  several  years  of  experimentation  with  3-D 
visualization  and  many  aspects  of  this  tool  are  in  hand.  Data  analysis  has  seen  movement 
of  two-dimensional  paper-based  graphics  in  the  seventies  to  the  computer  based  three- 
dimensional  color  graphics  of  the  eighties.  Three-dimensional  graphics  have  been 
achieved  by  kinematics  displays  where  motion  gives  the  depth  cues  and  by  stereoscopic 
displays.  We  have  used  both  red-green  stereo  and  polarized-light  stereo  to  good  effect. 
However,  as  satisfying  as  these  displays  are,  they  clearly  represent  the  paradigm  of  a 
scientific  investigator  looking  at  a  screen.  What  we  are  doing  now  is  to  leave  this 
scenario  and  immerse  the  scientific  data  analyst  in  a  data  world.  Allowing  the  analyst  to 
literally  move  about  in  the  data  world,  to  fly  around  the  data,  to  look  at  different 
portrayals  of  the  data,  to  make  a  turn  and  literally  turn  into  another  dimension  seamlessly. 
We  would  imagine  a  direct  manipulation  setting  where  an  analyst  can  actually  grab  hold 
of  a  data  point  in  this  virtual  world  and  see  the  effect  of  moving  it  around.  We  would 
imagine  a  progressive  disclosure  setting  in  which,  as  the  analyst  approached  a  data  point 
in  this  virtual  data  world,  the  numerical  value  or  other  attributes  of  the  data  point  would 
come  into  focus.  What  we  are  proposing  is  to  create  a  data  analysis  environment  where 
the  scientist  could  quite  literally  explore  the  data  much  the  same  way  that  an 
oceanographer  with  scuba  gear  or  a  submersible  could  explore  the  sea. 

Indeed,  this  undersea  exploration  analogy  is  a  good  one.  To  understand  the 
potential  impact,  one  can  imagine  an  oceanographer  who  wishes  to  explore  the  ocean.  On 
the  one  hand,  the  oceanographer  could  create  a  remotely  piloted  vehicle  with  a  television 


and  light  system  and  watch  the  exploration  process  on  a  video  monitor  within  a  darkened 
room  in  his  ship.  This  is  to  a  large  extent  what  computer-based  visualization  does  now.  A 
second  model  might  be  called  the  scuba  divers  model.  In  this  model,  the  oceanographer 
straps  on  a  scuba  tank  and  other  associated  gear,  dives  in  the  water  and  explores  the 
ocean  directly.  This  is  a  qualitatively  different  experience  and  has  historically  yielded  a 
different  type  of  result.  A  third  model  we  might  call  the  aquarium  model.  The  scuba 
divers  model  obviously  has  the  handicap  of  being  encumbered  with  clumsy  equipment 
and  such  distracting  concerns  as  running  out  of  air.  The  data  analytic  analog  has  the  same 
kind  of  concerns  with  wearing  a  tethering  helmet  apparatus  with  limited  visual  resolution. 
The  aquarium  model  is  an  intermediate  stage  and  is  analogous  to  visiting  an  aquarium 
with  a  40  foot  glass  wall.  We  have  implemented  a  20  foot  stereoscopic  projection  system 
which  will  allow  the  data  analyst  to  view  the  data  through  a  very  large  window 
unencumbered  by  requirements  to  wear  physical  hardware. 

Conceptually  what  we  wished  to  develop  is  easy  to  grasp.  We  proposed  a  human- 
computer  interface  system  which  will  allow  the  human  operator  control  of  his  or  her 
motion  through  the  virtual  data  world  simply  by  movement  of  an  instrumented  hand  or 
controller.  The  binocular  visual  and  binaural  audio  headgear  will  also  be  equipped  to 
sense  position  and  motion.  The  computations  to  update  views  are,  in  principle, 
straightforward  three-dimensional  Euclidean  geometry  computations.  The  basic 
limitations  tend  to  be  the  accuracy  and  resolution  of  the  input  devices  and  the  throughput 
of  the  rendered  images.  Systems  that  use  a  high-powered  graphics  workstation  for  each 
eye  still  run  below  30  frames  a  second  for  complex  images. 

The  popular  version  of  immersive  technology  typically  is  built  around  a  helmet 
system  which  incorporates  stereoscopic  video,  binaural  audio  and  position  and  motion 
sensors.  The  technological  weakness  of  these  systems  is  two-fold.  Because  of  the  close 
proximity  of  eyes  and  other  sensitive  organs,  the  use  of  CRT  displays  seems  inadvisable. 
Most  helmet  systems  have  been  based  on  liquid  crystal  technology  which  has  severe 
limitations  on  visual  resolution.  A  second  issue  for  such  systems  is  the  latency  of 
response  to  the  motion  sensing.  There  are  numerous  anecdotal  reports  on  nausea  induced 
by  the  latency.  Both  of  these  problems  are  technological  problems  which  can  be  expected 
to  be  improved  in  the  future.  We  have  somewhat  preferred  the  stereoscopic  projection 
system  as  an  immersive  technology  because  it  is  currently  capable  of  a  much  higher 
resolution,  and,  using  a  six  degree  of  freedom  controller,  capable  of  steering  the  image 
with  much  less  latency.  The  effect  is  that  the  room  essentially  becomes  a  vehicle  with  a 
portal  to  the  virtual  environment.  Both  of  these  immersive  technologies  offer  positive  and 
negative  aspects,  and  both  will  be  explored  in  the  context  of  this  present  proposal. 
Indeed,  we  envision  the  situation  that  both  can  be  used  simultaneously  by  different 
individuals  in  the  same  virtual  environment. 

Results  of  the  Project 

This  project  had  numerous  positive  results,  some  anticipated  when  the  proposal 
was  written  and  some  not  anticipated.  Results  generally  fall  into  three  categories: 
immersive  methods  results,  visualization  results  and  genered  statistical  results.  The  papers 
and  technical  reports  listed  below  indicate  the  extent  of  our  productivity.  Individual  titles 
convey  a  sense  of  what  was  accomplished  in  each  of  these  three  categories.  Some  results 


particularly  worth  mentioned  include  the  use  of  the  our  visualization  technique  called 
Mode  Forest  applied  to  ARL/SLAD  ballistic  vulnerability  data  and  the  visualization 
techniques  we  called  Saturation  Brushing  applied  to  the  classification  of  NSWCC/DD 
chemical  warfare  data. 


L2  Distance  to  Random  Point 

Figure  1.  Mode  forest  visualizing  the  ARL/SLAD  ballistic  vulnerability  data  for  the  Ml 
Abrams  tank.  The  two  dark  regions  to  the  right  of  the  image  around  3  and  5  indicate  two 
major  clusters  while  the  lighter  region  around  2  on  the  abscissa  is  a  smaller  cluster.  If 
printing  resolution  is  adequate,  there  are  two  subclusters  in  each  major  cluster. 

The  ballistic  vulnerability  data  consists  of  the  investigation  of  effect  of  behind 
armor  debris  on  67  subsystems  of  an  Ml  Abrams  tank  simulation.  Congressional 
requirements  for  live  fire  tests  create  a  dilemma  for  the  U.S.  Army.  Tanks  are  obviously 
extremely  expensive  targets  so  that  few  are  used  in  live  fire  settings.  Because  the  sample 
space  for  binary  data  on  67  subsystems  has  2^^  items  in  it,  it  is  unreasonable  to  expect  that 
the  live  fire  tests  will  coincide  with  the  outcomes  of  simulation.  The  dilemma  is  thus  that 
it  is  argued  the  simulations  are  no  good  because  they  do  not  correspond  to  the  live  fire 
tests.  Our  suggestion  was  to  cluster  the  67  dimensional  data  to  see  if  the  live  fire  tests  fall 
into  one  of  the  clusters.  This  strategy  has  proven  very  successful  because  in  fact  the 
damage  mechanisms  do  cluster  into  two  main  clusters  and  one  minor  cluster  with  two 
subclusters  in  each  of  the  major  clusters.  The  graphic  illustrating  this  is  included  as 
Figure  1. 

The  NSWC/DD  chemical  warfare  data  involves  the  measurement  of  intensity 
levels  of  color  patches  on  treated  paper  in  each  of  13  spectral  bands.  The  treated  paper  is 


intended  to  respond  to  both  chemical  and  biological  agents  and  the  nature  of  the  spectral 
response  is  indicative  of  the  type  of  agent.  While  the  13  dimensional  vector  is  a  good 
discriminator,  computations  involving  the  full  vector  are  too  slow  for  a  real-time 
response.  It  is  desired  to  find  simple  discriminators  that  can  be  computed  real-time  for  an 
early  warning  response.  The  graphic  for  this  is  given  in  Figure  2. 


Figure  2.  Desaturated  parallel  coordinate  plot  of  the  chemical  warfare  data  indicating 
that  spectral  bands  B9  and  BIO  alone  are  sufficient  to  discrimate  among  the  three  classes 
of  chem-bio  agents  and  even  provide  discrimation  of  subclusters  with  in  one  type. 


This  type  of  graphic  allows  us  to  conclude  that  spectral  band  B9  and  BIO  together 
form  discriminators  which  are  as  effective  as  all  thirteen  bands  taken  as  a  whole.  With 
reduced  dimensionality  the  discrimination  procedure  could  be  executed  real  time. 
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